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1.  INTRODUCTION 


Numerical  Mathematics  is  a  very  broad  field.  In  this  paper ,we~£ocus*~on- 

/ 

aspects  of  numerical  mathematics  whiA  are  related  to  computer  science.^  If 

I 

our  focus  were  on  numerical  mathed ivies  as  a  mathematical  subject,  we  would 
cover  rather  different  ground,  for  ’.hat  aspect  of  the  subject  has  a  vast  liter¬ 
ature  which  we  do  not  touch  on  h*. >.  € .  The  reader  interested  in  the  relation 
between  numerical  mathematics,  c-S/lied  mathematics,  and  computation  is  re¬ 
ferred  to  Cohen  [4] . 

■Computer  Science  has  stro  .  <  mathematical  components  such  as  automata 
theory,  computational  complex-  numerical  mathematics,  and  symbolic  mathe¬ 
matics.  However,  much  of  cc  if -ter  science  is  rather  close  to  engineering  in 
the  sense  that  t* e  creations  must  work  in  the  real  world.  Similiarly,  numerical 
mathematics  ran  jes  from  sti  '  Ugly  mathematical  areas  to  those  with  a  strong  en- 

< 

giner-ing  flavor. 

One  of  the  central  concerns  of  computer  science  is  algorithmic  analysis 
or  algorithmics  (Traub  [25,  p.  1J).  As  we  shall  see,  numerical  mathematics  deals 
with  algorithms  for  continuous  processes. 


-2- 


\ 

2.  '  HISTORICAL  PERSPECTIVE  '  ~  " 

The  roots  of  numerical  mathematics  are  ancient  dating  back  thousands 
of  years  (Davis  [7]).  Yet  modern  numerical  mathematics  as  a  separate  dis¬ 
cipline  dates  back  less  than  25  years.  The  modem  discipline  is  distin¬ 
guished  by: 

(1)  synthesis  and  analysis  of  algorithms  rather  than  a  focus  on 
synthesis  alone 

(2)  analysis  of  algorithms  using  finite  precision  numbers 

Some  have  dated  the  modern  field  back  to  1947.  That  year  marked  the 
establishment  of  an  Institute  for  Numerical  Analysis  at  UCLA  as  well  as  the 
publication  of  a  paper  by  Von  Neumann  and  Goldstine  [27]  on  .the  error  analysis 
of  matrix  inversion  using  finite  precision  arithmetic.  Indeed  it  had  been 
believed  (Householder  [11],  Traub  [24],  Wilkinson  [29])  that  1947  marked  the 
first  use  of  the  phrase  "numerical  analysis",  but  Businger  [6]  pointed  out  that 
the  phrase  had  been  used  by  Lagrange.  Nevertheless  1947  does  seem  to  be  a 
reasonable  year  to  mark  the  beginning  of  modem  numerical  mathematics,  and  in 

t  * 

1572  a  joint  SIAM-SIGNUM  Symposium  will  celebrate  the  25th  anniversary. 

Both  numerical  analysis  and  numerical  mathematics  are  used  to  denote  the 
discipline  although  the  latter  appellation  seems  to  be  superseding  the  former. 


3.^A  DEFINITION  OF  NUMERICAL  MATHEMATICS  . 

We  give  a  formal  definition  of  the  discipline,  as  it  is  generally  prac¬ 
ticed  today  and  then  suggest  how  the  scope  of  the  field  should  be  broadened. 

Numerical  mathematics  is  the  theory  arid  practice  of  the  efficient  calcula¬ 
tion  and  error  appraisal  of  approximate  solutions  of  continuous  mathematical 
problems-. 

An  equivalent  name  for  the  subject  is  the  Analysis  of  Continuous  Algorithms. 
Two  of  the  words  in  this  definition,  continuous  and  efficient,  warrant  some 
amplification.  By  continuous  we  mean  the  problem  input  and/or  output  belong  to 
a  continuum.  For  example,  the  solution  of  polynomial  equations  with  integer 
coefficients  belongs  to  numerical  mathematics  since  the  zeros  belong  to  a 
continuum.  The  solution  of  diophantine  equations  does  not  belong  since  the 
input  and  output  are  integers.  The  distinction  between  these  two  problems  is 
a  real  .one  dince  the  second  problem  is  combinatorial  in  nature  and  totally 
different  techniques  are  used. 

The  adjective  efficient  is  critical.  One  of  the  primary  differ¬ 
ences  between  mathematics  and  numerical  mathematics  is  that  mathematics  lacks 
the  concept  of  efficiency.  There  is  elegance  and  simplicity  of  proof,  but  not 
efficiency. 

Mathematics  distinguishes  between  constructive  and  non-constructiye  proofs. 
A  constructive  proof  may  be  used  as  an  algorithm  but  it  is  usually  not  an  ef¬ 
ficient  one.  For  example,  the  proof  of  the  theorem  that  continuous  functions 
on  a  closed  interval  may  be  approximated  arbitrarily  closely  by  polynomials  ia 
based  on  Bernstein  polynomials.  This  construction  is  not  used  in  numerical 
practice  since  it  may  require  a  polynomial  of  degree  ten  million  to  achieve 
eight-place  accuracy.  A  best-fit  polynomial  of  degree  perhaps  six  would 
achieve  such  accuracy. 
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The  adjective  continuous  serves  to  delineate  the  discipline  as  it  is 
generally  practiced  today.  However,  the  algorithmic  approach,  long  tra¬ 
ditional  in  numerical  mathematics,  is  useful  for  problems  where  the  data 
does  not:  belong  to  a  continuum.  Numerical  mathematicians  can  contribute  to 
the  study  of  such  problems. 

Before  leaving  our  discussion  of  the  extent  of  numerical  mathematics, 
we  should  mention  seme  neighboring  fields.  Discrete  mathematics  (also  called 
combinatorial  mathematics)  deals  with  discrete  data  (such  as  integers).  Sym¬ 
bolic  mathematics  deals  with  symbolic  data.  Numerical  mathematics,  discrete 
mathematics,  and  symbolic  mathematics  are  all  subfields  of  the  mathematics  of 
computation  which  consists  of  the  intersection  of  mathematics  and  computer 
science. 

For  the  remainder  of  this  paper,  vz  discuss  four  major  components  of 
numerical  mathematics.  They  are 

1.  Foundations 

2.  Synthesis  and  Analysis  of  Algorithms 

3.  Analysis  of  Error 

4.  Programs  and  Program  Libraries 

Others  might  choose  different  component*? .  Many  numerical  mathematicians 
believe  (2)  and  (3)  comprise  the  field.  We  believe  that  the  study  of  (1)  pro¬ 
vides  theoretical  underpinnings  while  (4)  makes  results  achieved  in  numerical 
mathematics  accessible  to  other  fields. 


*  i 


■  tr, 
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4.  FOUNDATIONS  ..  - -  '  l)'"  /  " 

}  ' 

We  discuss  two  examples  of  foundation  subjects:  finite  precision  number 
systems  ana  computational  complexity. 

a.  Finite  Precision  Number  Systems 

The  finite  precision  floating  point  numbers  are  the  numbers  used  in 
computation  and  their  properties  eve  quite  different  from  the  real  numbers  of 
mathematical  analysis.  We  give  two  single  illustrations: 

The  familiar  algebraic  laws  need  not  hold.  Thus  a  ^  b,  x  +  a  “  x  +  b 
may  be  simultaneously  true.  A  perceptive  discussion  is  given  by  Van  Wijngaarden 
[26  j.  Knuth  [14,  Vol.  2,  Chapter  4]  contains  a  wealth  of  material. 

Another  phenomenon  is  that  the  spacing  between  adjacent  numbers  is  not 

uniform.  Consider  eight  place  numbers.  The  nearest  neighbor  to  a  number  cf 

*8  "*15  8 

size  10  ie>  10  distant.  The  nearest  neighbor  to  a  number  of  size  10 

is  101  distant.  Since  10  >  tt,  the  calculation  of  the  sine  of  a  rounded  number 

g 

of  magnitude  10  is  meaningless  with  such  numbers. 

The  "real  numbers"  and  "real  analysis"  may  be  viewed  as  a -limiting  case 
of  finite  precision  numbers  and  numerical  mathematics.  In  this  sense  numerical 
mathematics  may  be  regarded  as  "richer".  For  example,  Taylor's  formula  is  a 
special  case  of  Newton's  interpolation  formula  from  which  it  follows  in  the 
confluent  limit. 

The  real  numbers  of  analysis  were  introduced  to  simplify  mathematics. 

They  are  an  abstraction  placed  on  a  firm  foundation  by  Dedekind,  Weierstrass 
and  other  analysts  of  the  last  century.  The  implications  of  real  numbers  for 
real  analysis  took  decades  to  develop.  The  implications  of  finite  precision 
numbers  for  numerical  mathematics  are  just  beginning  to  be  explored,  Brent  [3] 
and  Matula  [16]  are  doing  interesting  work  here. 


b.  Computational  Complexity 

We  emphasized  above  thecentral  role  that  efficiency  plays  in  numerical 
mathematics.  This  focus  on  efficiency  naturally  leads  us  to  study  what  are 
the  most  efficient,  that  is,  the  optimal  processes.  This  area  of  study  is 
contained  in  computational  complexity.  Some  of  the  most  exciting  current  work 
in  numerical  mathematics  lies,  in  this  area.  In  March  1972,  an  IBM  Symposium  on 
Complexity  of  Computer  Computations  summarized  recent  progress. [12].  In  [22] 
we  suggested  that  optimality  theory  for  algebraic  (or  combinatorial)  pro¬ 
cesses  be  called  algebraic  computational  complexity,  while  optimality  theory 
for  analytic  (or  continuous)  processes  be  called  analytic  computational 
complexity. 

Much, but  not  all,  of  algebraic  computational  complexity  deals  with  al¬ 
gorithms  for  problems  which  are  of  interest  to  numerical  mathematics.  Such 
problems  include  the  multiplication  of  matrices  and  the  evaluation  of  poly¬ 
nomials.  Although  these  processes  are  combinatorial,  they  are  very  common 


components  of  the  algorithms  of  numerical  mathematics.  A  survey  may  be 
found,  in  Borodin  [2]. 


Research  on  analytic  computational  complexity  dates  to  the  early  sixties 
and  predates  most  of  the  algebraic  results.  A  survey  may  be  found  in  Traub 
[22].  The  results  achieved  so  far  deal  almost  exclusively  with  optimality  of 
iteration  algorithms  for  the  calculation  of  fixed-points. 

We  cite  here  only  one  very  recent  result  due  to  Rung  [15].  Rung  defines 
a  multiplication  efficiency  index  for  any  multivariate  rational  iteration  which 
takes  into  account  the  rate  of  convergence  of  the  iteration  and  the  number  of 
multiplications  or  divisions  per  iteration  step.  He  proves  that  the  multiplica¬ 
tion  efficiency  is  always  bounded  by  unity. 


5.  SYNTHESIS  AND  ANALYSIS  OF  ALGORITHMS  S  /  '  '">/ 

This  has  been  a  core  area  for  numerical  mathematics  and  will  continue 
to  be  so.  It  is  relatively  easy  to  invent  algorithms  (synthesis).  Th.s  trick 
is  to  invent  good  ones  and  prove  they  are  good  (analysis).  Knuth  considers 
the  analysis  of  algorithms  a  central  one  in  computer  science  and  is  writing  a 
seven  volume  set  on  this  subject  [14].  He  is  not  attempting  to  include  the 
algorithms  of  numerical  mathematics. 

In  practically  every  problem  area,  the  best  algorithm  we  have  today  has 
been  Introduced  in  the  last  15  years.  Some  examples  are  the  QR  algorithm  of 
Francis  and  Kublanovskaya  with  improvements  by  Farlett,  Kahan,  and  Wilkinson; 
the  algorithms  of  Fletcher  and  Powell  and  their  co-workers  on  minimization; 
rational  extrapolation  algorithms  for  ordinary  differential  equations  due  to 
Bulifsch,  Gragg,  and  Stoer;  Golub's  analysis  of  Householder  Transformations 
for  least  squares  problems,  the  analysis  of  finite  element  methods  for  partial 
differential  equations  due  to  Ciarlet,  Fix,  Schultz,  Strang,  and  Varga;  the 
efficient  calculation  of  finite  Fourier  Series  by  the  Fast-Fourier  Transform 
of  Cooley  and  Tukey.  These  are  only  a  few  instances  of  the  great  progress 
that  has  been  made.  Even  when  the  algorithms  are  classical,  as  in  the  case 
of  Gaussian  elimination,  it  is  only  recently  that  a  complete  analysis  has 
been  performed. 

We  list  some  general  c  'iteria  for  a  good  algorithm: 

1.  The  algorithm  should  do  well  when  performed  in  finite  precision. 

2.  It  should  be  robust,  that  is,  it  should  degr^d«  gracefully  if  it 
is  used  on  a  problem  which  is  near  <hu  boundary  of  the  problem 
space  for  which  the  algorithm  is  designed. 


S.  The  hypotheses  under  which  the  algorithm  works  should  be  easily 
verifiable.  For  example,  since  the  symmetry  of  a  matrix  is  easily 
verified,  an  eigenvalue  algorithm  which  converges  for  any  symmetric 
matrix  is  desirable.  On  the  other  hand,  it  is  hard  to  verify  a 
hypothesis  on  the  positivity  of  a  second  derivative  over  an  interval. 

4.  The  algorithm  should  be  efficient  by  some  appropriate  measure. 

5.  The  algorithm  should  always  work  for  a  reasonable  “large"  class 
of  problems. 

Such  algorithms  are  natural  candidates  for  implementation  as  computer 
routines. 

In  certain  areas  we  are  now  at  the  stage  where  we  have  excellent  algorithms 
meeting  these  specifications.  A  major  recent  achievement  has  been  to  design 
such  algorithms  for  such  areas  as  systems  of  linear  algebraic  equations, 
algebraic  eigenvalues,  and  polynomial  zeros.  Elsewhere  we  have  a  long  way 
to  go.  Examples  are  most  non-linear  problems,  most  multivariate  problems  such 
as  constrained  and  unconstrained  optimization,  partial  differential  equations, 
ordinary  differential  equations.  For  some  problems  we  may  never  find  algorithms 
satisfying  our  five  criteria.  Gentleman  [8]  has  surveyed  numerical  algorithms 
and  selected  areas  for  which  good  algorithms  do  or  do  not  exist. 

Almost  without  exception  algorithms  have  been  created  for  sequential 
machines ♦  Only  recently  have  algorithms  been  designed  for  parallel  and 
vector  machines.  A  survey  of  work  on  parallel  algorithms  may  be  found  in 
Miranker  [17]. 
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6.  ANALYSIS  OF  ERROR 

An  Important  element  in  numerical  mathematics  is  the  appraisal  of  error. 
We  consider  both  mathematical  and  machine-aided  error  analysis. 

Although  mathematical  error  analysis  began  with  the  celebrated  paper  by 
Von  Neumann  and  Goldstine  [27]  published  in  1947,  it  was  not  until  the  mid 
to  late  fifties  that  activity  really  picked  up.  Extensive  bibliographies  may 
be  found  in  Rail  [20]. 

One  of  the  most  important  ideas  in  mathematical  error  analysis  is  that 
of  backward  error  analysis  proposed  by  Givens  TiO]  and  brilliantly  exploited 
by  Wilkinson  [28]  to  analyze  the  rounding  error  of  algebraic  processes.  In 
backward  error  analysis  we  ask  what  problem  have  we  solved  exactly  and  how 
far  is  this  problem  from  the  one  we  set  out  to  solve?  This  may  be  contrasted 
with  forward  error  analysis  where  we  ask  by  how  much  the  calculated  answer 
differs  from  the  true  answer.  Backward  analysis  is  often  easier  to  perform 
and  the  answers  are  often  more  useful.  Using  backward  analysis,  Wilkinson 
[30]  was  able  to  give  a  complete  a  priori  analysis  of  the  solution  of  a 
system  of  linear  algebraic  equations  by  Gaussian  elimination  with  pivoting. 

We  turn  to  machine-aided  error  analysis.  A  dream  has  been  automatic 
error  analysis  —  let  the  computer  do  the  work.  Papers  in  machine-aided  error 
analysis  first  appeared  in  the  mid  to  late  fifties.  An  annotated  bibliography 
is  given  by  Bright  [5]  and  extensive  references  appear  in  Rail  [20]. 

Special  arithmetics  are  tools  used  in  machine-aided  error  analysis. 

The  purpose  of  a  special  arithmetic  is  to  enable  us  to  appraise  the  error 
at  the  same  time  that  we  perform  the  calculation.  Examples  are  significance 
arithmetic  (Ashenhurs:  [1])  and  range  arithmetic  (Moore  [18]).  There  are 
some  serious  problems.  For  example,  a  difficulty  with  range  arithmetic  is 


that  the  intervals  in  which  the  answers  are  guaranteed  to  lie  increase  greatly 
in  size  as  the  computation  proceeds. 

Work  has  started  on  systems  for  automatic  error  analysis.  P.  Richman 
has  been  studying  the  limitations,  design  and  implementation  of  a  system 
for  computing  numerical  results  to  within  any  desired  tolerance.  His  basic 
tool  is  variable-pr  vision  range  arithmetic. 
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Stanford-Texas  joint  project  on  the  Wilkinson  matrix  codes,  and  the  1MSL  pro¬ 
gram  library  [13].  A  collection  of  recent  papers  on  mathematical  software  may 
be  found  in  the  proceedings  of  a  Purdue  conference  edited  by  Rice  [21]. 
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